Brain-computer interface with high-speed eye tracking features

ABSTRACT

Embodiments described herein relate to systems, devices, and methods for use in the implementation of a brain-computer interface that integrates real-time eye-movement tracking with brain activity tracking to present and update a user interface that is strategically designed for high speed and accuracy of human-machine interaction. Embodiments described herein also relate to the implementation of a hardware agnostic brain-computer interface that uses real-time eye tracking and online analysis of neural signals to mediate user manipulation of machines.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to and the benefit of U.S. ProvisionalPatent Application Ser. No. 62/549,253, entitled “Brain-ComputerInterface with High-Speed Eye Tracking Features,” filed Aug. 23, 2017,the disclosure of which is hereby incorporated by reference in itsentirety.

BACKGROUND

Embodiments described herein relate to systems, devices, and methods foruse in the implementation of a brain-computer interface that integratesreal-time eye-movement tracking with brain activity tracking to presentand update a user interface that is strategically designed for highspeed and accuracy of human-machine interaction. Embodiments describedherein also relate to the implementation of a hardware agnosticbrain-computer interface that uses real-time eye tracking and onlineanalysis of neural activity to mediate user manipulation of machines.

A brain-computer interface (BCI) is a hardware and softwarecommunications system that permits brain activity alone to controlcomputers or external devices with direct communication pathways betweena wired brain and the external device. BCIs have been mainly designed asan assistive technology to provide access to operating machines andapplications directly from interpreting brain signals One of the maingoals of BCI development is to provide communication capabilities toseverely disabled people who are totally paralyzed or ‘locked in’ byneurological neuromuscular disorders, such as amyotrophic lateralsclerosis, brainstem stroke, or spinal cord injury, for whom effectivecommunication with others may be extremely difficult.

Some known implementations of brain computer interfaces include spellerslike the one designed by Farwell and Donchin. In this speller, the 26letters of the alphabet, together with several other symbols andcommands, are displayed on-screen in a 6×6 matrix with randomly flashingrows and columns. The user focuses attention on the screen andconcentrates successively on the characters to be written, while theneural response of the brain is monitored for signature signals. Oncedetected the signature signals allow the system to identify the desiredsymbol. The Farwell-Donchin speller allows healthy people to spell atthe rate of about 2 characters per minute.

However, BCIs can be designed to assist even physically able people tooperate computers or other data-processing machines and/or softwareapplications without the need for conventional input or outputinterfaces such as a mouse and a keyboard. BCIs may also provide aninterface for more intuitive and natural interaction with a computerthan conventional input control. Additionally, BCIs can also bedeveloped to serve many other functions including augmenting, repairingas well as mapping and researching human and animal cognitive and/orsensory motor systems and their functions. Some BCI applications includeword processors, adapted web browsers, brain control of a wheelchair orneuroprostheses, and games, among others, However, most applicationshave solely been designed for training or demonstration purposes.

SUMMARY

Systems, devices and methods are described herein for variousembodiments of a hardware-agnostic, integrated oculomotor-neural hybridbrain computer interface (BCI) platform to track eye movements and brainactivity to mediate real-time positioning of a user's gaze or attentionand selection/activation of desired action. This disclosure presents anintegrated hybrid BCI system to address the need for Brain ComputerInterfaces that operate with high-speed and accuracy.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a schematic illustration of a hybrid Brain ComputerInterfacing system, according to an embodiment.

FIG. 2A is an illustration of a video based eye-tracking device.

FIG. 2B shows a front and rear view of a video based eye-trackingdevice.

FIG. 3A is an illustration of an exemplary neural recording headset.

FIG. 3B shows a side view (left) and a top view (right) of exampleelectrode placement locations over the scalp of a user for recordingneural brain signals.

FIG. 3C shows example responses recorded from the brain, indicating thedifference between Event Related Potentials (ERPs), a type of neuralactivity, elicited by a desired target stimulus (black) as opposed to anon-target stimulus (magenta), presented at the time point indicated bythe vertical, dashed, line.

FIG. 4 is a schematic illustration of an example hybrid BCI system usingan example stimulus driver neural activity (a P300signal), todiscriminate stimuli and select or deselect a stimulus based on userintent.

FIGS. 5A and 5B are illustrations of an example user interface beforeand after a user selection.

FIG. 5C illustrates a user interacting with the interfaces in FIGS. 5Aand 5B through a video based eye-tracker and a neural recording headset.

FIGS. 5D and 5E show example signals generated and recorded by the videobased eye-tracker and the neural recording headset monitoring brainactivity shown in FIG. 5C.

FIG. 5F illustrates the operation of a high-speed eye-movement trackingsystem that can be implemented to initiate and implement a desiredaction even before the subject consciously perceived their decision.

FIG. 6 is a schematic illustration of a hybrid Brain ComputerInterfacing Device, according to an embodiment.

FIG. 7 shows an example process of operation of a hybrid Brain ComputerInterfacing Device, according to an embodiment.

FIG. 8A shows an example set of brain signals recorded over time from aset of neural recording electrodes, using a hybrid BCI system accordingto an embodiment.

FIG. 8B shows an example set of neural brain signals recorded over timeusing a hybrid BCI system according to another embodiment.

FIG. 9A illustrates instances of an example user interface before (left)and after (right) a user selection, according to an embodiment.

FIG. 9B illustrates instances of another example user interfacefollowing another layout, before and after user selection, according toanother embodiment.

FIG. 10 shows several instances of an example user interface,illustrating the operation of dynamic tagging of a moving object intime.

FIG. 11 shows an illustration of a space of utilization of BCI systemsbased on user capabilities and the type of BCI system.

DETAILED DESCRIPTION

Embodiments described herein relate to systems, devices, and methods foruse in the implementation of a hybrid brain-computer interface (BCI)that integrates real-time eye-movement tracking with brain activitytracking to present and update a user interface that is strategicallydesigned for high speed and accuracy of human machine interaction.Embodiments described herein also relate to the implementation of ahardware agnostic brain-computer interface that uses real-time eyetracking and online analysis of neural brain signals to mediate usermanipulation of machines.

For BCI technology to be better suited for patients, useful to thegeneral public, and employed in the control of real-world tasks, theinformation transfer rate has to be improved to meet a naturalinteractive pace, the error rate has to be reduced, and the complexityof the interaction interface has to be minimized, compared to currentimplementations. Additionally, BCI applications demand a high cognitiveload from the users, thus the user interface has to be improved to moveaway from quiet laboratory environments into the real world. In order toconfigure BCI devices and applications to be easier and more intuitive,there exists a need for improved devices and techniques in theimplementation of brain machine interfaces that operate with high-speedand high accuracy to enable user mediated action selection through anatural intuitive process.

A Hybrid BCI System

As described herein, a BCI is a hardware and software communicationssystem that permits brain activity alone to control computers orexternal devices. A hybrid BCI system includes a display of stimulithrough an interface, a hardware apparatus to locate the point of focusof a user on the interface, a device for recording and processing brainactivity, and an apparatus for effecting control of the interface, whichmay translate into control over the user's environment. These standardfeatures can be characterized as (1) a pointing control feature, (2) anaction control feature, and (3) a user interface feature. The pointingcontrol feature can be analogized to a conventional pointing device likea mouse pointer that allows a user to narrow down to a small set ofmanipulators to control. The action control feature can be analogized toa selection device, for example a mouse click or a keyboard stroke, thatallows the user to effect change with an action. The action can be oneof many, for example, an activation or a deactivation, a continuouschange to the user interface (e.g. scrolling) or an acute change to theuser interface with discrete starts and stops (e.g. highlighting,hovering, etc.), pinching, zooming, titling, rotating, swiping, amongothers. Some other examples of action control via a user interface caninclude a virtual keyboard control, menu navigation, actions to placeand unplace object or items, action to move objects or items, expandand/or shrink objects, movement or navigation of the first personobserver or player, changing perspectives of the observer, and actionslike grabbing, picking or hovering. Some of these aspects of actioncontrol are disclosed below. The user interface feature in a hybrid BCIsystem can be analogized to an operating system that creates andmaintains an environment that implements the pointing and action controlfeatures in addition to other features like offering a selection menu,navigation controls, etc.

In some embodiments of a hybrid BCI system described herein, thepointing control feature and methods for identifying a user's point offocus can include an eye-tracking device. In some embodiments, theaction control feature and methods for identifying the intent of theuser can include any suitable form of monitoring neural signals in thebrain. This can include, for example, brain imaging through electricalor optical methods. For example, in some embodiments, the hybrid BCIsystem can use electrodes recording neural signals of brain activity,channeled through an amplifier and a processor that convert the user'sbrain signals to BCI commands. In some embodiments, the hybrid BCIsystems can implement sophisticated user interfaces that implement brainactivity based control of machines. Specific adaptations to one or moreof these features can be implemented, as described in detail below, toachieve high speed and accuracy of human interaction with the hybrid BCIsystem.

FIG. 1 is a schematic illustration of a hybrid Brain Computer Interfacesystem 100, according to an embodiment. The example hybrid BrainComputer Interface system 100 (also referred to herein as “hybrid BCIsystem” or “BCI system” or “system”) is an integrated oculomotor-neuralhybrid BCI system that includes a video based eye-tracker 102 and aneural recording headset 104 for recording one or more control signalsof the user's brain. The control signals can be any form of neuralactivity recorded through any suitable approach likeelectroencephalography (EEG), electrocorticography (ECoG) ormagnetoencephalography (MEG), etc. Example forms of neural activityinclude Event Related Potentials (ERPs), motor imagery, visually evokedPotentials (VEP), brain state dependent signals, etc. The example hybridBCI system 100 also includes a Brain-Computer Interfacing Device 110,and optionally an audio-visual display 106.

In some embodiments of the hybrid BCI system 100, the user's point offocus can be determined from the video based eye-tracker 102 (i.e., thepointing control feature), and the neural recording headset 104 can beused for collecting neural brain activity data (i.e., the action controlfeature). The collected neural and eye-tracking data can be communicatedto the Brain-Computer Interfacing Device 110 that processes the signalsas an ensemble along with data. about what stimuli were presented. Withthe combined information, the Brain-Computer interfacing Device 110 candetect relevant signal features based on statistical models to predictthe user's intent. This predicted intent can then be communicated to theuser, via the user interface presented through the display 106 forexample, and acted upon.

Video Based Eye-Tracking—The Pointing Control Feature

In some embodiments, the video based eye-tracker 102 can be used todetermine where a user is looking in their visual field by rapidlyfollowing the eye movements of the user in a two or three dimensionalspace. For example, provided the user has voluntary control of theireye-movements, the video based eye tracer 102 can be used to determinewhich subspaces in their visual field their eyes are “pointing to.” Inother words, the video based eye-tracker 102 can use the user'seye-movement trajectories as a pointing control feature, revealingsignificant information about the subject's intent and behavior. In someembodiments, aspects of where in the visual space their attentionfocused, what stimulus they are focused upon, or what stimulus theyresponded to, can be used effectively in the BCI system 100.

In some embodiments, the video based eye-tracker 102 relies on trackingthe user's pupil and a first-surface corneal reflection (CR) of anillumination source with the use of a head-mounted eye tracking videocamera to image the user's eye. The positional difference between thesetwo features can be used to determine the observer's eye-in-headorientation. Some example head mounted eye-tracking devices that can beused as the video based eye-tracker 102, are shown in FIGS. 2A and 2Band are available from SenseMotoric Instruments, Tobii Eye Tracking, andPupil-labs among other commercial vendors. For example, the illustrationin FIG. 2B shows a front and rear view of a video based eye-tracker,with the left and right eye cameras indicated. The cameras can beconnected to each other, through a wired or wireless connection. Thevideo based eye-tracker can also include an additional scene camera thatcaptures the uses' field of view.

In some embodiments, the video based eye-tracker 102 simultaneouslyrecords an image of the pupil and the corneal reflection of anillumination source. The video based eye-tracker 102 can use a nearinfrared (IR) illumination source that is optimally reflected by theiris and is invisible to humans so it does not disturb or distract theuser. The strong IR reflectance yields high contrast images that areparticularly beneficial to pupil detection. In some embodiments, thevideo based eye-tracker 102 can use a collimated, far range light sourcewhereby parallel rays are emitted from a distant illumination source andcollimated by optical components. In some embodiments, the video basedeye-tracker 102 can use a non-collimated near-source for illuminatingthe eye whereby the illumination source is mounted at a finite distance(typically 50 mm or less) from the eye and there is no optical componentbetween the source and the eye to collimate the rays.

As described herein, the video based eye-tracker 102 utilizes the lightreflected from the eye, which is sensed by a video camera or some anyother suitable optical sensor specially designed for this use. Thesensed light is then analyzed to extract eye rotation from changes inreflections. In some embodiments, the video based eye-tracker 102 canuse the corneal reflection (i.e., the first Purkinje image) and thecenter of the pupil as features to track over time. In some embodiments,the video based eye-tracker 102 can use reflections from the front ofthe cornea (i.e., the first Purkinje image) and the back of the lens(i.e., the fourth Purkinje image) as features to track eye movement in amore sensitive approach. In some embodiments, the video basedeye-tracker 102 can use even more sensitive methods of tracking byimaging features inside the eye, such as, for example, the retinal bloodvessels, and following the movement of these features as the eyerotates.

In some embodiments, the video based eye-tracker 102 can include anintegrated display as described below. The video based eye-tracker 102integrated with a display 106 can be a system configured to view virtualreality space. In some embodiments, the video based eye-tracker 102integrated with a display 106 can be configured to view augmentedreality space.

Neural Recording of Brain Signals—The Action Control Feature

Central to any BCI system are brain signals that can be used as controlsignals, making the brain signals an action control feature.Electrophysiological activity, one of the main types of brain activitiesto be monitored, is generated by electro-chemical transmittersexchanging information between the neurons. The neurons generate ioniccurrents which flow within and across neuronal assemblies. The largevariety of current pathways can be simplified as a dipole conductingcurrent from a source to a sink through the dendritic trunk. Theseintracellular currents are known as primary currents. Conservation ofelectric charges dictates that the primary currents are enclosed byextracellular current flows, which are known as secondary currents.

As described herein, the neural recording headset 104 uses brain signalsto gather information on user intentions through a recording stage thatmeasures brain activity and translates the information into tractableelectrical signals that can be converted into commands. In someembodiments, the neural recording headset 104 can be configured torecord electrophysiological activity through electroencephalography(EEG) which has a high temporal resolution, low cost of set-up andmaintenance, high portability, and is non-invasive to users. The neuralrecording headset 104 includes a set of electrodes having sensors thatacquire electroencephalography signals from different brain areas. Thesesensors measure electrical neural activity caused by the flow ofelectric currents during synaptic excitations of the dendrites in theneurons and is extremely sensitive to the effects of secondary currents.The neural signals are recorded through the electrodes appropriatelyarranged in the neural recording headset 104 and placed on the scalp ofa user. An example illustration of an embodiment of the neural recordingheadset 104 is shown in FIG. 3A, and similar neural recording headsetsare available from Biosemi, Wearable Sensing and G.Tec among othercommercial vendors.

In some embodiments, the neural recording headset 104 can includeelectrodes, amplifiers, AID converter, and a recording device. In someembodiments, the electrodes on the neural recording headset 104 canacquire the signal from the scalp and the amplifiers can magnify theanalog signals to enlarge the amplitude of the neural signals. In someembodiments, the one or more electrodes arranged in the neural recordingheadset 104 can be directly connected to amplifiers, A/D convertors, andone or more recording devices to store signals from each electrode. Insome embodiments, these components can be housed in the neural recordingheadset 104. In some embodiments, only the immediate signalamplification can be carried out in the neural recording headset 104 andthe other processes like A/D conversion and recording can be carried outafter transferring the signal to B-C Integrating Device 110.

In some embodiments, the electrodes on the neural recording headset 104can be arranged to be placed over the scalp based on the commonlyfollowed International 10-20 system, standardized by the AmericanElectroencephalographic Society. FIG. 3B shows an example illustrationof the placement of electrodes according to this system. The 10-20system uses two reference points on the head to define the electrodelocation. One of these reference points is the nasion, located at thetop of the nose at the same level as the eyes. The other reference pointis the inion, which is found in the bony lump at the base of the skull.The transverse and median planes divide the skull from these two points.The electrode locations are determined by marking these planes atintervals of 10% and 20%. The letters in each location correspond tospecific brain regions in such a way that A represents the ear lobe, Cthe central region, Pg the nasopharyngeal, P the parietal, F thefrontal, Fp the frontal polar, and O the occipital area. FIG. 3C showsan example set of brain responses following a stimulus, so calledstimulus driven “Neural Activity”, recorded using the neural recordingheadset 104. A neural signal is measured as the potential differenceover time between an active electrode (also referred to as a signalelectrode) and a reference electrode. In some embodiments, a thirdelectrode, known as the ground electrode, can be used to measure thedifferential voltage between the active and the reference electrodes. Insome embodiments, the neural recording headset 104 can include one ormore active electrodes, one or more reference electrodes, and one groundelectrode. In some embodiments, the neural recording headset 104 caninclude a few as seven active electrodes. In some embodiments, theneural recording headset can include up to 128 or 256 active electrodes.The electrodes can be made of silver chloride (AgCl) or any othersuitable material. The electrodes can be configured so that theelectrode-scalp contact impedance can be appropriately adjusted torecord an accurate signal. The electrode-tissue interface can beresistive as well as capacitive and it therefore behaves as a low passfilter. The impedance depends on several factors such as the interfacelayer, electrode surface area, and temperature.

Neural signals recorded non-invasively, across the scalp of a user, haveto cross the scalp, skull, and many other layers which can make themweak and hard to acquire. Neural signals can also be affected bybackground noise generated either within the brain or externally overthe scalp, which can impact the ability to extract meaningfulinformation from the recorded signals. Embodiments of the system 100including the neural recording headset 104 can incorporate severaladaptations to improve neural signal acquisition. For example, a gel(i.e., a conductive gel) can be used to create a conductive path betweenthe skin and each electrode to reduce the impedance. In someembodiments, the neural recording headset 104 can include “dry”electrodes that do not need the use of gels, which can be made withother materials such as titanium and stainless-steel. In someembodiments, the neural recording headset 104 can include dry activeelectrodes with pre-amplification circuits to accommodate the very highelectrode/skin interfacial impedances. In some embodiments, the neuralrecording headset 104 can include dry passive electrodes that do nothave any active circuits, but may be linked to a neural recording systemconfigured with ultra-high input impedance.

The amplitude of electrical bio-signals is typically on the order ofmicrovolts. Consequently, the signal is very sensitive to electronicnoise. In some embodiments, the BCI system 100 can be designed to reducethe effects of the noise with adaptations such as electromagneticinterference shielding or reduction for common mode signal, amongstothers.

Display and Presentation of the User Interface

As described herein, the user interface in the hybrid BCI system 100functions as a link of communication between the user (e.g., the user'sbrain, eyes, etc.) and the BC Interfacing Device 110, and enables a userto focus and point at specific stimuli through the pointing controlfeature and select or deselect specific stimuli using the action controlfeature. A strategically designed user interface includes a process ofpresentation of stimuli to a user through any modality, and mostcommonly though the visual mode. For example, user interfaces can beconfigured to present a user with visual stimuli, auditory stimuli,haptic stimuli or vestibular stimuli. In some embodiments, a userinterface that presents visual stimuli can be rendered on a display likethe display 106 shown in FIG. 1 .

In some embodiments, the display 106 can be a separate, stand-alone,audio-visual display unit that can be connected and in datacommunication with the rest of the hybrid BCI system 100. That is, astand-alone display (e.g., a liquid crystal display) equipped with anaudio system (e.g., speakers, or headphones) can be in two-waycommunication with one or more of the other components of the hybrid BCIsystem 100, for example, the BC Interfacing Device 110, the video basedeye-tracker 102, and the neural recording headset 104. In someembodiments, the display 106 can be integrated into the video basedeye-tracker 102 to be part of the eye-glass area. The integrated videobased eye-tracker 102 and display 106 can be configured to view virtualreality space in the form of a user interface presented on the display106. In some embodiments, the integrated video based eye-tracker 102 anddisplay 106 can be configured such that the display 106 is on asemi-transparent eye-glass area, allowing the user to view augmentedreality space. That is, the user can view the real-world through thesemi-transparent eye-glass area that is also the integrated display 106presenting the user with a user interface that he/she can interact with.

The Brain-Computer Interfacing Device

In some embodiments, the BC Interfacing Device 110 can be configured toaccomplish three main functions. First, the BC Interfacing Device 110can be configured to generate a strategically designed user interface.For example, the strategically designed user interface can be for atraining session or for a testing session. In some embodiments, the userinterface can be designed as a virtual reality interface and/or as anaugmented reality interface. In some embodiments, the user interface canbe tailored for specific needs such as, for example, specific userhistory, reaction times, user preferences, etc. Second, in addition todesigning and generating the user interface, the BC Interfacing Device110 can be configured to receive the pointing control signal (e.g., fromthe video based eye-tracker 102) and the action control signal (e.g.,from the neural recording headset 104) and processes the signals as anensemble to determine the user's intent. Finally, the BC InterfacingDevice 110 can be configured to implement the pointing control featureand the action control feature by (1) detecting meaningful features fromthe neural signals, and (2) implementing changes to the stimuli beingpointed to per the user's intent. In some embodiments, the BCInterfacing Device 110 can also connected to other peripheral devicesthat may be a part of the hybrid BCI system 100 such as, for example,peripheral sensors and actuators functioning in modalities other thanthe visual modality. Such peripheral sensors may include audiomicrophones, haptic sensors, accelerometers, goniometers etc., andperipheral actuators can include audio speakers, haptic stimulusproviders, etc.

In some embodiments, the BC Interfacing Device 110 can include anInput/Output Unit 140 that receives and sends signals from the videobased eye-tracker 102, the neural recording headset 104, and theoptional audio visual display 106 through one or more data communicationports. The transfer of signals can also be carried out through a wiredconnection, or wirelessly through any suitable communication channellike Bluetooth, etc. The functions of the Input/Output Unit 140 caninclude several procedures like signal acquisition, signal preprocessingand/or signal enhancement, etc. The acquired and/or pre-processed signalcan be channeled to a processor 120 within the BC Interfacing Device110. In some embodiments, the processor 120 and its sub-components (notshown) can be configured to handle the incoming data, send and retrievedata to and from a memory 160, and carry out the functions of buildingand maintaining a user interface which can be rendered on the display106 or on a display integrated with the video based eye-tracker 102. Insome embodiments, the processor 120 and its sub-components can beconfigured to carry out the functions needed to enable user-specificinterpretation of brain signals, and packaging output signals to theInput/Output Unit 140 to be relayed to external devices. Other functionsof the processor 120 and its sub-components can include severalprocedures like feature extraction, classification, and manipulation ofthe control interface.

Control Signals in the Hybrid BCI System

As described herein, the purpose of the hybrid BCI system 100 is tointerpret user intentions by monitoring cerebral activity and locating acontrol signal that can be used as the action control feature. Theformation of user intent can be denoted as a cognitive task. Theintegrated hybrid BCI system 100 shown in FIG. 1 can use one of severalsignature brain signals simultaneously evoked by or related to cognitivetasks performed by a user. Some of these brain signals can be decoded inways that people may learn to modulate them at will. Using thesesignals, regarded as control signals, can enable the hybrid BCI system100 to interpret the intentions of the user. Thus, the hybrid BCI system100 can record and use control signals from the brain and use them as anaction control feature controlling the interface between the brain andthe computer. Any suitable neural activity can be a control signal usedfor implementing the action control feature. Some examples of neuralactivity in time include Event Related Potentials (ERPs), EvokedPotentials (EPs e.g. visually evoked potentials (VEP), auditory evokedpotentials, sensory evoked potentials, motor evoked potentials), motorimagery signals, slow cortical potentials, brain state dependent signalsand other, as yet undiscovered, signature activity potentials underlyingvarious cognitive or sensorimotor tasks.

As an example form of neural activity, an Event Related Potential or anERP can be a signature neural activity related to an event or a stimuluspresentation correlated in time. ERPs can have distinct shape andfeatures (like the P300 signal known to peak at around 300 ms followingthe triggering stimulus) that helps with their detection andidentification. ERPs can also vary in size and shape across differentbrain regions and how they map across brain regions can be indicative ofspecific brain functions and/or user intentions. The neural activitydata acquired from the neural recording headset can be analyzed forspecific ERP signals and once detected and classified appropriately theBCI Device 110 can implement any particular action associated with thedetected ERP on the desired portion of the user interface.

Another example control signals can be the form of Motor imagery signalswhich are neural activity signals associated with the user undergoingthe mental cognitive process underlying voluntary or involuntary motion.That is, motor imagery signals are brain signals that can be recordedfrom various brain regions and analyzed by a BCI system 100 while theuser imagines the action and/or performs the action. In embodimentswhere the users are expected to make the bodily movements or gestures,the BCI system can also use information gathered by a set of peripheralsensors coupled to the BCI Device 110, such as goniometers,torsiometers, accelerometers, etc., to help recognize the gesture inhigh definition during a training session. In some other embodiments,users may not be required to make the physical movements, as it may besufficient to imagine a set of movements to evoke the motor imagerysignals (e.g. after training sessions where a BCI system is allowed tolearn associations between a set of movements of a user and a set ofmotor imagery signals evoked by the movements).

Yet another example of control signals can be in the form of brain statesignals or brain state dependent signals. For example, a BCI system canbe used to detect patterns or scalp topographies of EEG signalsassociated with specific attentional states or perceptual states of thebrain of a user. Neural activity that can be used by a BCI system canalso be the frequency domain. Some examples among others includesensorimotor rhythms, Event Related Spectral Perturbations (ERSPs),specific signal frequency bands like Delta, Theta, Alpha, Beta, Gamma orMu rhythms etc. One example, the P300 evoked potential, is describedbelow.

P300 Signals

P300 evoked potentials are positive peaks in the neural signal due toinfrequent auditory, visual, or somatosensory stimuli. The P300 signalis an “endogenous” brain signal that depends on the processing of thestimulus context and levels of attention and arousal. Without wishing tobe bound by any particular theory, the P300 is widely believed to be aneural signature of the mechanisms required to change the mental modelof the environment to make an appropriate response. Thus, the P300signal is an important signature of cognitive processes such asattention and working memory. Advantageously, the use of P300-based BCIsdoes not require training.

Endogenic P300 responses are elicited about 300 ms (up to 900 ms) afterattending to statistically improbable stimuli or what are known as anoddball stimuli among several frequent stimuli. P300 signals can belocated over several brain regions and can be indicative of severalbrain processes. Recordings of brain activity associated with subjectsperforming controlled tasks with visual targets have shown theoccurrence of these consistent, signature signals to be associated withspecific stimuli. Target-related responses can occur in the parietalcortex and the cingulate areas of the brain and novelty-relatedactivations can mainly occur in the inferior parietal and prefrontalregions. Stimulus modality-specific contributions come from the inferiortemporal and superior parietal cortex for the visual and from thesuperior temporal cortex for the auditory modality.

This large (>10 μV) positive deflection can occur in participants whenthey are uncertain about the nature of the upcoming stimulus. The P300is commonly associated with “oddball” paradigms, in which occasionalrelevant (“target”) stimuli have to be detected in a train of frequentirrelevant “non-target” or “standard” stimuli. The hybrid BCI system 100can use any neural activity like the P300 signal to determine therelevance of a particular stimulus discriminating it from the others,and to determine the user intent when focusing on that stimulus.

An example P300 signal evoked by a particular target stimulus is shownin FIG. 3C (black trace) compared to the lack of a distinct signal whenanother non-target stimulus was presented (magenta), recorded using anexample neural recording headset 104 of an example system 100. Anoddball paradigm presented through an example user interface of thesystem 100 can reliably yield P300 responses with a parietocentral scalpdistribution to target compared to standard stimuli irrespective ofstimulus (visual, auditory, somatosensory) or response (button press,counting) modality. P300 responses can also be observed when trains ofregular stimuli are interrupted by stimulus omissions, which underlinesthe endogenous nature of this component.

The amplitude of the P300 can increase with lower probability and higherdiscriminability of target stimuli. The latency of P300 signals withrespect to the onset of the target stimulus can increase when targetstimuli are harder to discriminate from standard stimuli but not whenresponse times increase for other reasons. P300 latency is thus anattractive tool to separate the mental chronometry of stimulusevaluation from response selection and execution. The occurrence of P300signals and their association with specific stimuli can be utilized bythe system 100 as an action control feature to select the P300 elicitingstimulus over the non-eliciting one.

FIG. 4A shows a flowchart illustration of an example instance of how aneural activity can be used as a action control feature during theoperation of the hybrid MI system 100, according to an embodiment. Insome embodiments, the P300 signal can the neural activity used todetermine and act upon a user's intent to select one stimulus overanother by distinguishing the first from the second stimulus (as achange or difference). The presence and absence of a P300 signal in therecorded responses can be used to make the selection.

In some embodiments, the hybrid BCI system 100 can be used to manipulatethe presentation of various stimuli on the display 106 through the userinterface and determine the users intent based on the recorded brainsignals over several brain regions following stimulus presentation.Neural activity can vary in shape and size over various brain regions,For example, the parietocentral P300 or P3b is elicited by task-relevantdeviant stimuli that are attended to, deviant stimuli that areirrelevant for the task but more salient than the targets elicit aslightly earlier positive deflection that has a frontocentral scalptopography. This novelty P300 or P3a component can be elicited byinserting unique and highly salient (“novel”) stimuli in the trains ofrepeated standard and target events (e.g., visual fractals in trains ofsimple geometrical figures). However, a similar P3a component can alsobe elicited by highly discrepant repeated distractors in paradigms thatuse a difficult target/standard discrimination.

VEP Signals

Visually evoked potential (VEP) signals are brain activity modulationsthat occur in the visual cortex after receiving a visual stimulus. Thesemodulations are relatively easy to detect due to the large amplitude ofVEPs as the stimulus is moved closer to the central visual field. In thehybrid BCI system 100 disclosed herein, VEPs can be used in conjunctionwith the eye-tracking input, to adopt an integrated approach to enablingthe pointing control feature.

VEPs can be classified as transient VEPs (TVEPs) and as steady-stateVEPs (SSVEPs), TVEPs occur when the frequency of visual stimulation isbelow 6 Hz, while SSVEPs occur in reaction to stimuli of a higherfrequency. TVEPs can be elicited by any change in the visual field, forexample flashing lights, patterns turned on or off, or abruptly reversedin their contrast. TVEPs vary with the stimulus presented and are rarelyused in BCI systems.

SSVEPs, more commonly in use with BCI systems, are elicited by visualstimuli changing at a frequency higher than 6 Hz. If the stimulus is aflash, SSVEP shows a sinusoidal-like waveform, the fundamental frequencyof which is the same as the blinking frequency of the stimulus. If thestimulus is a pattern, the SSVEP occurs at the reversal rate and attheir harmonics. In contrast to TVEP, constituent discrete frequencycomponents of SSVEPs remain closely constant in amplitude and phase overlong periods of time. SSVEPs are less susceptible than TVEPs toartifacts produced by blinks and eye movements and to electromyographicnoise contamination.

Hybrid BCI System: The Integrated Approach

As described herein, video based eye-tracking can be used in conjunctionwith neural activity in the form of VEPs or ERPs or other forms ofneural activity to enable the pointing control feature and using neuralactivity as control signals (for example ERPs, or sensorimotor signals,motor imagery signals, brain state signals, etc.) to enable the actioncontrol feature. The BC Interfacing Device 110 can be configured toprocess the integrated signals as an ensemble. In some embodiments, thehybrid BCI system 100 can use SSVEP signals from the neural recordingheadset 104 in addition to the eye-tracking signal from the video basedeye-tracker 102 to have an integrated, hybrid approach to locating thepoint of gaze of a user. That is, in some embodiments, the video basedeye-tracker 102 can be used to detect any suitable form of eye movementinformation, for example, saccadic, foveation and/or pupil dilationinformation, as well as foveation information, through oculomotor dataconveying movement of the eye muscles. Information about saccadic eyeposition can also be indirectly obtained from neural activity, forexample, ERPs that are evoked by visual responses, acquired from theneural recording headset 104. For example, the hybrid BCI system 100 canbe configured to correlate the occurrence of ERPs with the presentationof a particular stimulus in time and space to form a causalrelationship. Thus, oculomotor data from the video based eye-tracker 102can be combined with data from visually evoked neural activity and thestrategic presentation of stimuli in the user interface deliveredthrough the display.

An integrated hybrid approach to tracking eye movements allows a user torapidly select a target by voluntary movement of their eye-gaze, withsignals from the video based eye-tracker 102 and the visually evokedneural activity complementing each other in the information provided tolocate gaze, as the pointing control feature. The user visually fixesattention on a target and the BC Interfacing Device 110 can identify thetarget through features analysis of the neural activity, and featureanalysis of the video based eye-tracker signal. Notably, in the hybridBCI system 100, the signals from the video based eye-tracker 102 and theneural recording headset 104 are analyzed as an ensemble, withappropriate weighting for each signal source, by the BC InterfacingDevice 110.

For example, the hybrid BCI system 100 can utilize visually evokedneural activity elicited by particular stimuli that may be desired bythe user, recorded by the neural recording headset 104. The userinterface presented on a visual display (e.g., the display 106) caninclude a matrix of symbols or commands in specific arrangements. Forexample, the symbols can be arranged in rows and columns and the rows orcolumns of this matrix can be flashed at random while the neural signalsare monitored. As the user gazes at the desired symbol, the neuralactivity can be elicited only when the row or column containing thedesired symbol flashes. The hybrid BCI system 100 can use data from thevideo based eye-tracker 102 (i.e., the eye-movement data) and data fromthe neural recording headset 104 (i.e., the stimulus presentation data)to implement a pointing control feature in order to triangulate andrapidly locate and determine the target symbol desirable to the user.

The hybrid BCI system 100 can use an ensemble processing technique todetermine the desired target stimulus. The ensemble processing techniquecan simultaneously process signals from one or more sources ofinformation available, including eye position, stimulus presentation,neural activity and brain signals associated with visual responses byweighting them appropriately. Upon locating and determining the desiredtarget symbol, the hybrid BCI system 100 can use the recorded neuralactivity (e.g., a P300 signal) to implement the action control feature.

For example, in some other embodiments of the hybrid BCI system 100,operating under one or more modes, the action control feature can beimplemented to select stimuli based only on the stimulus presentation atthe time that a neural activity is elicited, regardless of the positionof the eye position at the time. In this mode, the weighting of signalsmay be skewed in favor of the stimulus presentation information duringensemble processing.

Alternatively, in some embodiments, the action control feature can beimplemented in one or more modes wherein the hybrid BCI system selectsany feature that the users gaze may be upon, at the time-point when aneural activity is detected (accounting for suitable user response timesand reaction delays). Thus, any symbol or stimulus that the user's gazepasses over when the neural activity is detected will be selectedregardless of which stimulus may have caused the neural activity to beelicited. This feature can be implemented by altering the weights infavor of the eye-position signal to determine which stimulus to selectduring ensemble processing. Embodiments of the hybrid BCI system 100 caninclude operational modes capable of both scenarios presented above andother intermediate modes that can also operate in between these examplemodes by suitably altering the weighting given to the various signalsused, for example, the eye-position signals, the stimulus presentationinformation (including properties of the stimulus, spatiotemporalaspects of presentation, sequence of presentation, etc.) and the neuralactivity recorded. The hybrid BCI system 100 can also use other sourcesof information including biological models of the eye of the user,likelihood estimates of a particular neural activity being generated bya particular stimulus based on prior information or cumulativeinformation gathered in real-time).

In some embodiments, the arranged symbols can be flashed several timesin different combinations to sample the user's choice multiple times.The signals from each combination can then be averaged to improve theaccuracy of detecting the correct symbol. Visual symbols can beaccompanied by stimuli of other modalities like auditory or hapticstimuli.

In BCI systems generally, repetition of trials can be cumbersome andslow down the process of selection using neural activity signals likethe P300. However, reducing repetitions may risk lowering the accuracyof the selection. In the hybrid BCI system 100 described herein, theaccuracy of neural activity-based selection can be improved whilemaintaining few repeated trials by using sophisticated classifiers asdescribed below. The detection accuracy of visually evoked neuralactivity and the user experience in general can also be improved byoptimizing the properties of stimulus presentation (e.g. display ofvisual symbols) and by choosing and varying appropriate properties ofstimuli (e.g. dimensions or colors of the symbols, their arrangementetc.). Experience can also be improved by configuring the user interfaceto use specific feedback from the user to predict behavior.

One of the advantages of using the video based eye-tracker 102 inaddition to neural activity from the neural recording headset 104 isthat the eye movement can be analyzed very rapidly while maintainingrobust signal acquisition, By implementing sophisticated analyticalroutines the position of a user's gaze and focus can be processed inreal-time. Further, the video based eye-tracker 102 may not besusceptible to the same noise sources as neural activity recordedthrough the neural recording headset 104. Processing eye movements usingboth neural activity as well as video based eye-tracker signals in anintegrated approach as described herein can provide a more robust signalamenable to the high-speed and high-accuracy interface that isdesirable. In this configuration, one channel can compensate for theweaknesses of the other. Furthermore, the approach of processing bothdata sets as an ensemble allows the appropriate weighting of theindividual signals according to other parameters like user history andspecific details of the interface navigated, etc.

In some embodiments, the oculomotor response can be used to changestimulus presentation sequences. Visually evoked neural activity whilehaving a high accuracy of eye movement detection require repeatedflashing of stimulus. This may have unpleasant effects on the userexperience. To avoid this the hybrid BCI system 100 can, for example,estimate the quadrant of focus of the user from the oculomotorinformation and reduce or restrict the number of visual stimuli thathave to be flashed to that quadrant of the display. Thus flashing ofsubsections of the display can be linked to a saccade and flashing canbe triggered only after a saccade to that location.

User Interaction with the Hybrid BCI System

FIGS. 5A-5E illustrates a user's interaction with the hybrid BCI system100, according to an embodiment. In this example, the hybrid BCI system100 is being used to spell words in a two-step process, and the display106 presents several sub-groupings of characters (e.g., letters, numbersand symbols commonly found on a keyboard) in FIG. 5A. The user wears avideo based eye tracker 102 and a neural recording headset 104, shown inFIG. 5C. When the user focuses their gaze on a sub-group containing thedesired letter (e.g., the sub-group indicated by the highlighted circlein FIG. 5A), the user interface presented in the display 106 changes tothat shown in FIG. 5B where focused sub-group is magnified. The user canthen perform the action of selecting a letter by focusing their gaze onthe specific desired letter in that sub-group. The action controlfeature is then implemented by using the neural activity recorded toperform the selection of a letter to be used in forming a word or asentence.

The pointing control feature described above with reference to FIGS. 5Aand 5B, is implemented with the video based eye-tracker 102 shown inFIG. 5C. The video based eye-tracker 102 is configured to detect wherethe user is focusing their gaze, and then output a signal as shown, forexample, in FIG. 5D. The action control feature (i.e., activation) isimplemented with the neural recording headset 104 shown in FIG. 5C. Theneural recording headset 104 is configured to record neural signals fromthe user's brain, and then output a signal as shown, for example, inFIG. 5E. A processor (not shown) can then extract meaningful featuresfrom the eye-tracking signal (FIG. 5D) and the neural signal (FIG. 5E)as an ensemble, and analyze them either in an unsupervised and/orsemisupervised manner or by classifying the signals based on priormodels built through rigorous training with each specific user. Theanalyzed data can then be used to make predictions of user behavior suchas the point of focus of the user and/or the selection or activation ofa symbol upon which focus is predicted.

FIG. 5F shows an example flow of events in time following video-basedeye tracking. In some embodiments, as shown in the example in FIG. 5F,the saccadic eye movement can be detected by the hybrid BCI system evenbefore the conscious perception of self-movement by the user. That is,eye movements can be detected online as they occur immediately after oreven before the user's conscious decision. For example, the user may befocused on one grouping of letters as shown in FIG. 5A and may begin tomake an eye movement towards another grouping. Even before theyconsciously realize their own movement a hybrid BCI system disclosedhere can detect and use the eye movement to adapt the user interface asis appropriate, which in this case is magnifying the next grouping ofletters.

An Example Hybrid BCI System

FIG. 6 shows a hybrid BCI system 600 according to an embodiment. In someembodiments, the BCI system 600 can be similar in structure and/orfunction to the corresponding portions of the hybrid BCI system 100described above with reference to FIG. 1 . For example, the BCI system600 includes a video based eye-tracker 606, an neural recording headset604, an optional display 606, and a Brain-Computer Interfacing Device610 that can be the same or similar to the video based eye-tracker 106,the neural recording headset 104, the optional display 106, and theBrain-Computer Interfacing Device 110 of the hybrid BCI system 100.Accordingly, such similar portions and/or aspects are not described infurther detail herein.

In some embodiments, a processor 620 of the Brain-Computer InterfacingDevice 610 can include a Synchronized Event Logger 622 that receives theeye-tracking and neural signal data via an I/O unit 640, and timestampsthe signal data to be synchronous with each other. In some embodiments,the Synchronized Event Logger 622 can be configured to perform highspeed eye movement classification.

Gaze signals can be decomposed into saccadic (short duration) andfoveation (longer duration) periods. These different components of gazecan be better utilized if the temporal lag inherent in the currentlystandard threshold-based saccade detection algorithms can be overcome.In addition, video based eye-trackers can be associated withnon-physiological artifact, in particular related to blinks, and cameramovement with respect to the eye. To overcome these limitations amongothers, in some embodiments the Synchronized Event Logger 622 canincorporate a probabilistic filtration system with custom machinelearning algorithms to allow high speed eye movement detection whileminimizing eye-tracker camera sampling rate requirements. Specifically,the Synchronized Event Logger 622 can implement a lag-less filtercombined with a machine learning classifier that requires no explicittraining and can detect saccades in real-time within 2 microseconds ofmovement initiation.

Typically, any smoothing function over a times series (e.g., Butterworthfilter, moving average, filters) as implemented by currently availableBCIs introduce a lag in the smoothed values compared to the actualmeasured values, preventing a high speed real-time gaze classificationsystem. In some embodiments, the Brain-Computer Interfacing Device 610as described herein can use a non-linear generative model that estimatesgaze as a dynamical system, with state and measurement models based ongaze kinematics and technical parameters of video based eye-trackers. Insome embodiments, dual estimation, based on the dual Kalman filter, canbe used in order to estimate hidden states of noisy gaze data andapproximate the underlying system model. Such an implementation canallow a filtration of gaze data without introducing temporal lag.Additionally, a theoretical gaze kinematics model can be used toestimate missing gaze data when camera sampling rate is lowered, or whentransient artifacts occur. Simultaneously, this results in a machinelearning classification system that is automatically trained without theusers' explicit knowledge, detecting eye movements in real-time duringthe movement but not after.

In some embodiments, the Synchronized Event Logger 622 can be configuredto implement ensemble estimation of attention using disparatephysiological signals as described in examples of the integratedapproach. In some embodiments, the Synchronized Event Logger 622 can beconfigured to implement an analytical pipeline that uses: (1) a Bayesianlinear discriminant system, (2) state-of-the-art preprocessing, (3)spatial filtering, (4) a bagging ensemble classifier algorithm, and (5)a higher-order oracle algorithm that incorporates information from theclassification algorithm with program routines during the experimentaltask, to improve selection accuracy. The Brain-Computer InterfacingDevice 610 can also be configured to use parallelization andasynchronous processing of separable components in the pipeline toensure performance on consumer-level personal computers.

In some embodiments, the Synchronized Event Logger 622 can be configuredfor real-time processing of the oculomotor-neural data in the collectedensemble to detect user intent as described herein. For example, eachvisual stimulus presented can evoke a signature neural activity (e.g., aP300 signal) captured by the neural recording headset 604. As describedherein, the neural activity can act as a control signal indicating auser's attention to a stimulus and intent to act, and the probability ofovert attention at a particular visual stimulus or tag can be estimatedaccording a neural activity classification score for each visualstimulus or tag. There can be a decay in the signal associated with astimulus that reflects spatial uncertainty as well as overt attentionbased on the degrees in visual space centered on gaze fixation,according to the receptive field of the high fidelity foveal region inthe retina. Quantification of the neural activity score and the decay insignal can be used for real-time, accurate oculomotor-neural recordingbased user intent classification, solving a machine learning challengeof combining disparate physiological signals, as well as temporal driftsin neural activity signal latencies, over time. Furthermore, itmaximizes signal information, such that individual visual stimulus andgaze position signals can update the probability of overt attention formultiple objects.

In some embodiments, the processor 620 can include a Trainer 624 that isconfigured to present a training environment rendered through the userinterface. The training environment can be configured to present a userwith a set of predetermined controlled stimuli and record the ensuingeye movement and/or brain activity. This set of controlled stimuli andthe evoked eye and brain activity corresponding to each of thecontrolled stimuli can then be stored in a memory 660 and used by aModel Builder 626 as training data to build statistical models that aretailor made for individual users. The Model Builder 626 can use one ormore statistical tools like dimensionality reduction methods, featureextraction methods, machine learning tools to build classifiers etc. Themodels can be built, tested and cross-validated using the training dataprovided by the Trainer 624 and then can be used with new data from thatparticular user to achieve a high accuracy and speed of interaction withthe user interface.

In some embodiments, the processor 620 can include a predictor 628 thatcan receive new data from a particular user, classify the data based onresults from the model builder 626, and generate a prediction for userbehavior using statistical tools like maximum likelihood estimation,maximum a posteriori estimation, etc. In some embodiments, the processor620 can also include a Architecture Developer 632 that receives theeye-tracking and brain signal data as well as data from the othersub-components of the processor 620 (e.g., the trainer 624, the modelbuilder 626, and the predictor 628). The Architecture Developer 632 maynot be intended for real-time use, but for robust statistical analysesoff-line towards prototyping potential BCI algorithmic detectionarchitectures.

Usage and Implementation of a Hybrid BCI System

FIG. 7 shows an example process 700 of operation of an integrated hybridBrain Computer Interfacing Device, according to an embodiment. In someembodiments, the process 700 includes an initial step 701 of initiatingdata acquisition and pre-processing for a particular user associatedwith a neural recording headset and an eye-tracker (and other peripheralsensors and/or actuators) at a time-point. This initiation and signalacquisition can be carried out by, for example, the Synchronized EventLogger 622. The process 700 also includes a step 703 of applying atraining environment implemented by a trainer like the trainer 624described above. The presentation of the training environment caninclude a set of pre-programmed, controlled stimuli displayed in step705 and the ensuing eye movement and brain activity can be recorded instep 707. The presentation of varied, but controlled stimuli andcollection of corresponding eye-movement and brain data can be repeatedas shown by step 719 for a sequence of stimuli. The data can be fed intoa model builder (e.g., the model builder 626), and the model can beupdated with each set of new stimuli in the training set. Followingwhich, the process 700 can include the presentation of a new environmentthat contains one or more novel stimuli, but may be associated with oneor more of the preprogrammed stimuli that the system was trained with.Using the statistical models built by the model builder 626, thepredictor 628 can then use any suitable statistical tool to generatepredictions of user behavior like determining the point of focus in step713 and estimating the user intent (i.e., to select or activate aparticular symbol displayed) from the brain data in step 715. Based onthe predictions, the process 700 then includes a step 717 to realize theuser's intent in the user interface. For example, the step 717 caninclude selection of a letter in a speller, or selection of a characterin a game, or the selection of ON functionality associated with a TVsystem that can be operated in an augmented reality system.

Hybrid BCI System Adaptations Modes of Operation

The hybrid BCI systems 100 and/or 600 and other embodiments describedherein can be configured as exogenous BCI systems that use the brainactivity elicited by an external stimulus such as VEPs, auditory evokedpotentials (AEPs), or P300 signals. Exogenous systems do not requireextensive training since their control signals from neural activity canbe easily and quickly set-up. Notably, the signal controls can berealized with fewer neural recording channels and can achieve a highinformation transfer rate of up to 60 bits/min.

In some embodiments, the hybrid BCI systems 100, 600 can be adapted tooperate as endogenous BCI systems that are based on self-regulation ofbrain rhythms and potentials without external stimuli. For example,through neuro feedback training, the users learn to generate specificbrain patterns which may be decoded by the BCI such as modulations inthe sensorimotor rhythms. One advantage of an endogenous BCI is that theuser can operate the BCI at free will and make or imaging voluntarymovements that translate to movement in a pointing control feature likea cursor to any point in a two-dimensional space. Other features likehand gestures can also be trained with and used as cues to gainoperational control over a user interface relieving any constraints onthe user being restricted to the choices presented.

In some embodiments, the hybrid BCI systems 100, 600 can be used eitherin a synchronous (cue-paced) mode or in an asynchronous (self-paced)mode. Synchronous modes in BCIs can be of simpler design focusing on apredefined time window of brain activity triggered by a specificstimulus. In the synchronous mode the hybrid BCI system 100, 600 cananalyze only the brain signals during the predefined time window and anybrain signals outside the window can be ignored. Therefore, the user isonly allowed to send commands during specific periods determined by thehybrid. BCI system 100, 600. One advantage of a synchronous mode is thatthe onset of mental activity is known in advance and associated with aspecific cue. Users may also inadvertently perform blinks and other eyeor bodily movements, which can generate artifacts. As the brain signalsoutside a specified time window are not analyzed the misleading effectsof these artifacts are avoided.

In some embodiments, the hybrid BCI systems 100, 600 can operate in theasynchronous mode, which continuously analyzes brain signals regardlessof when the user acts. While more complex and hence computationally moredemanding, the asynchronous mode can offer a more natural mode ofhuman-machine interaction since the user does not have to wait for anyexternal cues. In addition, richer data is gathered as the userinteracts with the hybrid BCI system 100, 600 during the ordinary courseof his or her actions.

Implementing High-Speed Signal Detection and Classification

The oculomotor-neural integrated hybrid BCI system described hereinintelligently and flexibly exploits the strengths of oculomotor signalslike signals from the video based eye-tracker 102 and neural signalsincluding VEPs as selection and activation signals, respectively. Onekey component of this system is the high speed classification of eyemovements. As described above with reference to the operation of theSynchronized Event Logger 622 the hybrid BCI system 600 can beconfigured to perform saccade detection in a lagless manner. Forexample, the hybrid BCI system 600 can filter gaze data with zero lag,using a non-linear generative model that estimates gaze as a dynamicalsystem, with state and measurement models based on gaze kinematics andtechnical parameters of video based eye-trackers. Dual estimation, basedon a dual Kalman filter, can be used in order to estimate hidden statesof noisy gaze data and approximate the underlying system model. Thus,filtration of gaze data can be accomplished without introducing temporallag. In addition, theoretical gaze kinematics models can estimatemissing gaze data when camera sampling rate is lowered, or whentransient artifacts occur. The hybrid BCI system 600 can also include amachine learning classification system that is automatically trainedwithout an explicit training session enabling real-time detection of evemovements.

In some embodiments, lagless saccade detection can translate into anability to detect saccade onset while eye movement is in progress. Asshown, for example, in FIG. 5F, the detection latency (approximately 2ms after saccade onset) can be shorter than a human's perception of theshift in visual scene (approximately 50-150 ms). Therefore, the hybridBCI systems 100 and/or 600 can use the saccade onset vector as aprediction of where a user will look, and then update the userenvironment accordingly, before the user is aware of the shift in visualinformation. For example, during a program or a game, the user may planto pause the game. As the user's eyes begin to shift direction towards apause menu item, the saccade onset vector can be used to predict theintent to pause and select the pause menu before the user shift gaze tofocus on the pause menu item. In some embodiments, the detection latencycan be approximately 1.00 ms, 2.00 ms, 3.00 ms, 5.00 ms, 10.0 ms or15.00 ms (or any time period there between).

Treatment of Artifacts

Artifacts are undesirable signals that contaminate brain activity andare mostly of non-cerebral origin. Since the shape of neurologicalphenomenon is affected, artifacts may reduce the performance ofBCI-based systems. Artifacts can be physiological in origin ornon-physiological or technical artifacts.

Physiological artifacts are usually due to muscular, ocular, and heartactivity. Large disturbances in brain signals can arise from electricalactivity caused by muscle contractions, which occur when patients aretalking, chewing or swallowing. Other artifacts can be produced byblinking and other eye movements. Blinking makes generallyhigh-amplitude patterns over brain signals in contrast to eye movementswhich produce low-frequency patterns. Artifacts associated with cardiacactivity introduce a rhythmic signal into brain activity. Technicalartifacts are mainly attributed to power-line noises or changes inelectrode impedances, which can usually be avoided by proper filteringor shielding. Avoidance of physiological artifacts is difficult andrejection approaches suggest discarding the epochs contaminated by theartifacts. Automatic rejection may fail when artifact amplitudes are toosmall. Besides, rejection methodology leads to portions of theinteraction that the user loses device control when artifactcontaminated signals are discarded. Instead of rejecting samples, theartifact removal approach attempts to identify and remove artifactswhile keeping the neurological phenomenon intact. Common methods forremoving artifacts in neural are linear filtering, linear combinationand regression, BSS and PCA. Instead of removing artifacts fromrecordings of brain signals, the system 100 and/or 600 can processartifacts to offer a communication path that can be used.

In some embodiments, the hybrid BCI systems 100, 600 can deriveadvantages from the artifacts to the benefit of user experience. Forexample, the systems 100, 600 can be configured to detect artifactualmovements like eye blinks, jaw clenches, and other cardiac relatedchanges. Detecting and classifying these artifacts gives scope to usethe original neural signal for noise free applications. In someembodiments, the detected artifacts in the oculomotor-neural data canalso be taken advantage of due to their known causality in the user'sbehavior. For example, artifacts like eye-blinks, jaw clenches, andfacial expressions can be used as control signals to implement theaction control feature, as disclosed below.

Embodiments of the hybrid BCI systems 100, 600 operated in theasynchronous mode can use an unsupervised classification system thatincorporates a warm training set with a clustering model, andexamination of Euclidean distance within n-dimensional feature space.The artifact identification approach which can be carried out by theuses a sliding window of data, over which features are calculated andthen classified based on a clustering model over an n-dimensional space.Artifact intensity described over time can be used to classify throughany suitable cluster based technique, for example classification methodsbased on the Euclidean distance of a data point to each clustercentroid.

Conventional BCI systems record data and at set time intervals, extractand process the recorded signals to determine user intent and userstate. In some embodiments, the hybrid BCI system 100 and/or 600described herein can be configured to process recorded data with eachincoming sample to check for the presence of an “event” and using thepresence of the “event” or lack thereof to trigger a chain reaction bythe program. This can be further analysis of the data, changing theprocessing pipeline in anticipation of certain behaviors, or modifyingthe user's environment in some way. The process can be handled in amultithreaded environment by spinning off a processing pipeline on eachincoming sample (from any device source) and passing that data through aclassifier to determine its type.

Events include eye blinks, event potentials, movement artifacts,excessive signal noise, and sudden drops in observed activity. Followingthe analysis stage, the interface can determine if a classifiable eventhas occurred, and if so, it can trigger a specific sequence of events.An example epoch of activity recorded is shown in FIG. 8A. For instance,when an eye blink is observed, the trigger can instruct the system toignore incoming eye data for the next 5 milliseconds. When an eventpotential is observed, the trigger can instruct the system to identifythe correlation between environment changes and the sudden presence ofthe event to identify which specific change affected the user's brainactivity. Such example events are highlighted by the green and red barsin FIG. 8A.

Previous approaches to BCI implementation utilize a fixed time delay(i.e. 50 ms) after an event. After the fixed time delay passes, the userenvironment is changed, and the data pertaining to that time period isanalyzed. In contrast, the system 100 and/or 600 according to someembodiments can implement an approach where no specific time delays ortime periods are used, but instead incoming data is analyzed in acontinuous manner. When the system identifies and classifies a specifictype of event (as defined by the experimenter), the appropriate dataextracted can be analyzed for additional information. For example, whenan event is detected such as the one indicated in FIG. 8B, the detectioncan trigger a sequence of processes. Thus, the brain activity can act asa controlling signal and the detection of specific features can becomeactuators for additional events, actions and/or outcomes. This approachcan be analogized to performing an action like clicking a mouse orpressing a button which directly triggers an immediate action as opposedto a polling model where, for example, a user might update their mailingaddress, and a phone book periodically to replace the old address withthe new address.

User Interface Adaptations

As described herein, the user interface in the hybrid BCI system 100,600 can be presented through an audio-visual display (for example,display 106 in FIG. 1 ). The user interface enables a user to focus andpoint at specific stimuli through the pointing control feature andselect or deselect specific stimuli using the action control feature. Inorder to carry this out, a user interface includes a selection set whichis the group of available choices a user can make. Examples of selectionsets can include letters/numbers on an alphanumeric matrix, ordirectional arrows, or groups of distinctly shaped icons or targetsstrategically arranged on a control display. The selection methoddescribes how a command from a user will be interpreted by the BCI,either directly or indirectly. Direct selection allows a user todirectly select any item from the selection set, while indirectselection requires an intermediary step before a user can make aselection.

FIG. 9A shows two example layouts of a user interface including symbols(letters or numbers). In one example, all of the symbols are presentedand flashed together at a time. In the second example, only one symbol(X) is shown in the flashed group while the others are hidden in theform of asterisks. In some embodiment, this second layout can improveprocessing times with reduced perceptual problems known to decreaseneural activity based BCI classification accuracy. This layout may alsobe used to indicate a change in the user interface strategy, forexample, the layout may indicate the entry into a ‘hold-release mode’that is described further below.

The user interface can be configured to present the characteristics ofthe interface between the user and the BCI. The user interface caninclude three types of characteristics: (1) spatial, (2) sensory and (3)activation/deactivation. The spatial characteristics include thedimension, number and shape of the icons or targets. The sensorycharacteristics include the stimulus and feedback provided to the user,whether auditory, visual or somatosensory. The activation/deactivationincludes the quality of the human/technology interaction.

The quality of interaction can be characterized by the effort (howdifficult it is to use the BCI), displacement (how much movement isrequired to respond), flexibility (the number of ways in which the BCIcan be used), durability (how reliable the BCI hardware is),maintainability (how easily the BCI can be repaired) and the method ofactivation or release (the ability to make/activate or stop/deactivate aselection and how that selection is made). Activation and deactivationcan have distinct functionality, and using only activation as a actioncontrol input can be thought of as a trigger or momentary switch. Inthis case, only the activation causes an effect and the duration withwhich the activation is held does not alter the outcome. Using bothactivation and deactivation allows for more complicated controlfunctionality and the control input can act as a button. For example, ona remote control for a television, you can activate and hold one of thevolume keys to keep increasing the volume. In this case, holding aselection causes continued change, while releasing it keeps the currentstate.

Neural Activity Based Hold-Release Model

As described herein, the BCI systems 100, 600 can incorporate as part oftheir user interface a neural activity based BCI functionality in whichthe initial activation and the deactivation (hold-release) of targets ina neural activity based BCI can be separately controlled. This can allowthe hybrid BCI systems 100, 600 to be used in applications that requireindirect selection or applications that require quick changes betweenstates. Further, it can allow confirmation-cancelation of a selectedtarget by either holding the selection or switching attention to arelease target.

In a potential real-world application, the targets on the BCI displaycan have different activation/deactivation characteristics. An examplehybrid BCI user interface layout for a real-world application is shownin FIG. 8B. Some items can be hold-release enabled to allow fineadjustment, for example, reclining a wheelchair shown in the layout inFIG. 8B item B1, changing the temperature (item A3 and B3), orincreasing the volume of a television (items C2 and D2). Safety-criticalitems, such as unlocking/locking a door (items D1 and E1), could requirea hold-release confirmation-cancelation step, where a short hold periodis required before activation. The remaining items would performtraditional discrete neural activity based actions, such as turning onlights or changing a television channel (C1, A2 and B2). Once the userselects a target with a hold-release response (for adjustment orconfirmation) then the screen will change to a hold-release mode (FIG.9B, right panel), in which only the previously selected target and arelease target will be active and the rest of the targets on the BCImatrix will not be selectable. If the BCI correctly identified thedesired target, the user will hold the selected target and the BCI willperform the action either until the user wanted the action to stop(relining wheelchair or changing television volume) or for a specifiedduration to confirm the selection (thereby preventing inadvertentactivation of a safety-critical action). Thus, hold-releasefunctionality expands the utility of neural activity based BCIs in waysthat mirror the multiple control modes available on existing assistivetechnology.

During the holding process, the only information required by the BCI iswhen the user changes their selection (e.g., stops increasing/decreasingvolume or recline a wheelchair). The binary nature of the releasedecision allows the BCI to make the decision from very few flashesinstead of after multiple sequences of flashes. For the BCI user, thistranslates into a faster response time and a more continuous controlthan using the traditional neural activity based BCI approaches.

Neural Activity Based Mapping of Eliciting Stimuli

Traditional BCI systems (that observe how a user's brain activitychanges in response to changes made to their environment) generallyattempt to elicit all possible behaviors a user might have beforeanalyzing all the data obtained at once, and from there determining userintent. In some embodiments, the hybrid BCI system 100 and/or 600described herein can analyze the user's brain activity each time achange is made, and specifically using the information obtained fromthis analysis stage to inform the next change made to the environment.For example, if the user has 10 possible actions, then 10 changes can bemade to the environment to observe which change eliciting the most idealresponse (as defined by the program). Alternatively, in the hybrid BCIsystem 100 and/or 600, if the same user has the same 10 choices, theinformation obtained from the first change can be used to inform thenext change to be made, and pre-emptively eliminate certain actions,thereby reducing the duration it takes to go from intent to action.Thus, knowledge of how the user's brain responds to each change made tothe environment is not entirely required since each change informs onmore than just one potential user intent. This reduces the amount ofinformation to be recorded and instead intelligently determines whatinformation will be needed to identify user intent, obtains thatinformation quickly, and proceeds to creating the intended action.

In some embodiments, the hybrid BCI system 100 and/or 600, can alsoimplement error correcting codes to carry out feature extraction fromacquired neural signals to improve information transmission rates usingthe neural activity-based system. Additionally, the hybrid BCI systemcan utilize novel approaches to reduce refractory effects which are oneof the main sources of noise in a neural activity based BCI system themby superimposing the targets on a suitable background or by usingalternative stimulus types based on motion.

Dynamic Stimulus Detection and Tagging

In some embodiments, the hybrid BCI system 100 and/or 600 canincorporate dynamic stimuli or tag existing dynamic stimuli either in anaugmented reality or a virtual reality presented in its user interface.That is, the hybrid BCI system 100, 600 can identify a moving stimulusand dynamically tag that stimulus to track its movements in time. Anexample implementation of such a dynamical identification and tagging isillustrated in FIG. 9 .

The hybrid BCI system 100, 600 can detect the presence of a movingobject and place selection tags onto the moving object. The tags canflash over the track timeline and maintain colocation with the object.The user can then use BCI selection to trigger an action with respect tothat object. Alternatively, the hybrid BCI system 100, 600 can mainly beconcerned with the start or end of the track. A time window of neuraldata near the track start or end could be examined to see if a neuralactivity was elicited. This would indicate that the newly object may beof interest to the user and could then trigger some action. In someembodiments, eye-tracking information can be used to see if a saccadewas triggered by the objects initial movement to further bolster theassumption of object as interest prior to triggering the action. Such anaction can be the dynamic tag placement as described above in additionto any others.

Feature Extraction and Model Building

Various thinking activities (cognitive tasks) result in differentpatterns of brain signals. In some embodiments, the hybrid BCI system600 (and/or 100), can include a pattern recognition system thatclassifies each signal pattern into a class according to its features.The pattern recognition system extracts some features from brain signalsthat reflect similarities to a certain class as well as differences fromthe rest of the classes. The features can then be measured or derivedfrom the properties of the signals which contain the discriminativeinformation needed to distinguish their different types. As describedabove, the Model Builder 626 of the can use an array of statisticaltools to build statistical models for features extracted from datacorresponding with specific users, under specific user interfaces,coupled to specific hard ware peripheral, etc.

The information of interest in brain signals is typically hidden in ahighly noisy environment, and brain signals comprise a large number ofsimultaneous sources. A signal that may be of interest may be overlappedin time and space by multiple signals from different brain tasks. Forthat reason, in many cases, it is not enough to use simple methods suchas a band pass filter to extract the desired band power.

Brain signals can also be measured through multiples channels. Not allinformation provided by the measured channels is generally relevant forunderstanding the underlying phenomena of interest. Dimension reductiontechniques such as principal component analysis or independent componentanalysis can be applied to reduce the dimension of the original data,removing the irrelevant and redundant information. Computational costscan thereby reduced.

Brain signals are also inherently non-stationary, thus, in someembodiments it may be desirable to obtain time information about when acertain feature occurs. Some approaches that divide the signals intoshort segments can be used where the parameters can be estimated fromeach segment. One or more data transformations can be performedincluding methods like Fast-Fourier Transforms (FFT), Wavelet transformsor adaptive autoregressive components, or techniques like stationarysubspace analysis (SSA) can be performed to reveal the non-stationarytime variations of brain signals. SSA decomposes multivariate timeseries into stationary and non-stationary components. Multiples featurescan be extracted from several channels and from several time segmentsbefore being concatenated into a single feature vector.

Signal Classification and Prediction

The aim of the classification step in the hybrid BCI system 100, 600 isthe recognition of a user's intentions on the basis of a feature vectorthat characterizes the brain activity provided by the feature step.Classification algorithms can be developed via either offline, online orboth kinds of sessions. The offline session involves the examination ofdata sets, such as BCI competitions data sets, which are collected froman adaptive or closed-loop system. The statistics of the data may beestimated from observations across entire sessions and long-termcomputations may be performed. The results can be reviewed by theanalyst with the aim of fine-tuning the algorithms. In some embodimentsof the hybrid BCI system, the data can be processed online in a causalmanner evaluating a real-world. The analytical algorithms used can betested in an environment in which the user profile may change over timeas a result of changes in motivation or involvement of the user, forexample.

Classification algorithms can be calibrated by users through supervisedlearning using a labeled data set. However, brain signals as welleye-movement signals can be inherently non-stationary, with changes intheir distinguishing characteristics over time. For example, thepatterns observed in the experimental samples during calibrationsessions may be different from those recorded during the online session.Alternatively, progressive mental training of the users or even changesin concentration, attentiveness, or motivation may affect the brainsignals. In some embodiments, the hybrid BCI system 600 (and/or 100)described herein can use adaptive algorithms that can adapt to thechanges in the distinguishing feature.

In some embodiments, the hybrid BCI system 100, 600 can usesemi-supervised learning algorithms for detecting and classifyingsignals. The semi-supervised learning classifier can be initiallytrained using a small labeled data set, after which the classifier canbe updated with on-line test data. This approach may reduce trainingtime required by updating the classifier during an online session in acontinuous manner.

In some embodiments, the hybrid BCI system 100, 600 may use unsupervisedlearning or reinforcement learning or a combination of these methods todetect and classify brain signals and/or eye-movement signals.Unsupervised methods can be used to find hidden structures in unlabeleddata, in order to classify them. Some embodiments of the hybrid BCIdevice can use unsupervised methods that rely on techniques forco-adaptive learning of user and machine or covariate shift adaptation.Some embodiments can use reinforcement learning methods that are basedon distinguishing neural signals elicited when a subject is aware of anerroneous decision. These feedback signals can be used as learningsignals to prevent that error from being repeated in the future.

CONCLUSION

In summary, systems and methods are described herein for use in theimplementation of an integrated hybrid Brain Computer Interface operableby a user in real-time. The disclosed system includes an eye-movementtracking system to implement a pointing control feature and a brainactivity tracking system to implement an action control feature. Bothfeatures are implemented through the presentation of a user interfacestrategically designed to enable high speed and accurate operation.Additionally, the disclosed systems and methods are configured to behardware agnostic to implement a real-time hybrid BCI on any suitableplatform to mediate user manipulation of virtual, augmented or realenvironments. FIG. 11 shows an illustration of the usage space of theinventive hybrid BCI system and methods.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. Where methods described above indicate certain eventsoccurring in certain order, the ordering of certain events may bemodified. Additionally, certain of the events may be performedconcurrently in a parallel process when possible, as well as performedsequentially as described above

Where schematics and/or embodiments described above indicate certaincomponents arranged in certain orientations or positions, thearrangement of components may be modified. While the embodiments havebeen particularly shown and described, it will be understood thatvarious changes in form and details may be made. Any portion of theapparatus and/or methods described herein may be combined in anycombination, except mutually exclusive combinations. The embodimentsdescribed herein can include various combinations and/orsub-combinations of the functions, components, and/or features of thedifferent embodiments described.

1. An apparatus, comprising: a display configured to present aninteractive environment to a user; an eye-tracker configured to recordeye-movement signals generated by the user; a neural recording deviceconfigured to record neural signals generated by the user; aninterfacing device, operatively coupled to the display, the eye-tracker,and the neural recording device, the interfacing device including: amemory; and a processor operatively coupled to the memory and configuredto: receive the eye-movement signals from the eye-tracker, and theneural signals from the neural recording device; generate and present astimulus, via the interactive environment, to the user; process theeye-movement signals and the neural signals to determine a point offocus of the user, based on at least one of the eye-movement signals andthe neural signals; process the neural signals received from the neuralrecording device to determine an action intended by the user; andimplement the action intended by the user and update the presentation ofthe interactive user interface.
 2. The apparatus of claim 1, wherein:the eye-tracker is a video-based eye-tracker, the neural signals includeelectroencephalogram (EEG) signals including visually evoked potentials,motor imagery signals, Event Related Potentials (ERP), and brain statedependent signals, and the processor is further configured to integratethe eye-movement signals and the EEG signals to determine the point offocus of the user.
 3. The apparatus of claim 1, wherein: the neuralsignals include electroencephalogram (EEG) signals including at leastone of visually evoked potentials, auditory evoked potentials, motorimagery signals, Event Related Potentials (ERP), and brain statedependent signals.
 4. The apparatus of claim 1, wherein the stimulusincludes at least one of a visual, an auditory, and a haptic stimulus.5. The apparatus of claim 1, wherein the processor is further configuredto define a set of portions of the display, and present the stimulusrepeatedly, in one portion from the set of portions of the display,based on the determination of the point of focus of the user.
 6. Theapparatus of claim 1, wherein the processor is further configured toimplement a lag-less filter and a classifier to process the at least oneof the eye-movement signals and the neural signals, to determine thepoint of focus of the user.
 7. The apparatus of claim 1, wherein theprocessor is further configured to implement a dual Kalman filter and aclassifier to process the at least one of the eye-movement signals andthe neural signals, to determine the point of focus of the user.
 8. Theapparatus of claim 1, wherein the processor is further configured to:generate a kinematics model of a set of eye-movements of a simulateduser; identify a set of missing data points in the eye-movement signals;compute a set of replacement data points to replace the set of missingdata points in the eye-movement signals, based on the kinematics model;and incorporate the set of replacement data points to replace the set ofmissing data points and to generate updated eye-movement signals.
 9. Anon-transitory processor-readable medium storing code representinginstructions to be executed by a processor, the instructions comprisingcode to cause the processor to: generate an interactive user environmentthat can be manipulated, by a user, to perform a set of actions; definea set of stimuli that can be presented to the user via the interactiveuser environment; present at least one stimulus from the set of stimulito the user; receive, from an eye-tracker and a neural recording device,inputs related to the users behavior; determine, based on the inputsreceived and the stimulus presented, an action intended by the user; andimplement the action via the interactive user environment.
 10. Thenon-transitory processor-readable medium of claim 9, wherein the code toreceive inputs includes code to cause the processor to: extract, fromthe inputs received from the eye-tracker, eye-movement signals generatedby the user; extract, from the inputs received from the neural recordingdevice, neural signals generated by the user, the neural signalsincluding electroencephalogram (EEG) signals, the EEG signals includingvisually evoked potentials, motor imagery signals, Event RelatedPotentials (ERP), and brain state dependent signals; and process theeye-movement signals and the EEG signals using an integrated approach,to determine a point of focus of the user.
 11. The non-transitoryprocessor-readable medium of claim 9, wherein the code to receive inputsincludes code to cause the processor to: extract, from the inputsreceived from the neural recording device, electroencephalogram (EEG)signals generated by the user, the EEG signals including at least one ofvisually evoked potentials, auditory evoked potentials, motor imagerysignals, Event Related Potentials (ERP), and brain state dependentsignals; and process the EEG signals to determine a point of focus ofthe user.
 12. The non-transitory processor-readable medium of claim 9,wherein the code to cause the processor to determine the action intendedby the user includes code to cause the processor to: extract informationrelated to the presented stimulus from the set of stimuli; define andapply a set of weights to each of the inputs received from theeye-tracker, the inputs received from the neural recording device, andthe information related to the presented stimulus, to generate weightedrepresentations of each of the inputs received from the eye-tracker, theinputs received from the neural recording device, and the informationrelated to the presented stimulus; combine the weighted representationsto form an ensemble representation; and process the ensemblerepresentation to generate a prediction of the action intended by theuser.
 13. The non-transitory processor-readable medium of claim 9,wherein the code to cause the processor to determine the action intendedby the user includes code to cause the processor to: extracteye-movement signals from the inputs received from the eye-tracker; andimplement a lag-less filter and a classifier to determine a point offocus of the user, based on the eye-movement signals.
 14. Thenon-transitory processor-readable medium of claim 9, wherein the code toreceive inputs includes code to cause the processor to: extracteye-movement signals from the inputs received from the eye-tracker;identify a set of missing data points in the eye-movement signals;generate a kinematics model for eye-movements of a simulated user;estimate a set of replacement data points to replace the set of missingdata points in the eye-movement signals, based on the kinematics model;and incorporate the set of replacement data points to replace the set ofmissing data points to generate updated eye-movement signals.
 15. Thenon-transitory processor-readable medium of claim 9, wherein the code toreceive inputs includes code to cause the processor to: extracteye-movement signals from the inputs received from the eye-tracker;identify artifactual data points in the eye-movement signals; generate akinematics model for eye-movements of a simulated user; estimate a setof replacement data points to replace the artifactual data points in theeye-movement signals, based on the kinematics model; and incorporate theset of replacement data points to replace the artifactual data points togenerate updated eye-movement signals.
 16. A method, comprising:presenting, to a user, a stimulus via an interactive user interface;receiving, from an eye-tracker and a neural recording device, inputsassociated with the users behavior; receiving information related to thepresented stimulus; generating an ensemble set of data including theinputs associated with the users behavior and the information related tothe presented stimulus; processing the ensemble set of data to determinean action intended by the user; and implementing the action via theinteractive user interface.
 17. The method of claim 16, wherein thepresented stimulus includes a dynamic, visual-stimulus, the methodfurther comprising: detecting a movement of the dynamic,visual-stimulus; tracking the movement of the dynamic, visual-stimulus;and tagging the dynamic, visual-stimulus.
 18. The method of claim 16,further comprising: extracting eye-movement signals from the inputsreceived from the eye-tracker; extracting neural signals from the inputsreceived from the neural recording device, the neural signals includingelectroencephalogram (EEG) signals, the EEG signals including visuallyevoked potentials, motor imagery signals, Event Related Potentials(ERP), and brain state dependent signals; and integrating theeye-movement signals and the EEG signals to determine a point of focusof the user.
 19. The method of claim 16, further comprising: extractingeye-movement signals from the inputs received from the eye-tracker; andimplementing a lag-less filter combined and a classifier, based on theeye-movement signals, to determine a point of focus of the user.
 20. Themethod of claim 16, further comprising: extracting eye-movement signalsfrom the inputs received from the eye-tracker; identifying a set ofmissing data points in the eye-movement signals; generating a kinematicsmodel to predict eye-movements of a simulated user; estimating a set ofreplacement data points to replace the set of missing data points in theeye-movement signals, based on the predicted eye-movements of thesimulated user by the kinematics model; and incorporating the set ofreplacement data points to replace the set of missing data points and togenerate updated eye-movement signals. 21-22. (canceled)